Skip to content

HDFS-15180.Split FsDatasetImpl lock to block Pool level#3889

Closed
MingXiangLi wants to merge 1 commit intoapache:trunkfrom
MingXiangLi:trunk-split-lock
Closed

HDFS-15180.Split FsDatasetImpl lock to block Pool level#3889
MingXiangLi wants to merge 1 commit intoapache:trunkfrom
MingXiangLi:trunk-split-lock

Conversation

@MingXiangLi
Copy link
Contributor

@MingXiangLi MingXiangLi commented Jan 16, 2022

  • DataNode performance bottleneck is mainly in datasetLock.DatasetLock not only influence performance and also influence stable.
  • We have split datasetLock to block pool level and volume level.And have stable run in 30000+ DataNode server in our cluster over than one year.It has big improvement for our client read write performance and stable.
  • This pr is split lock to block pool level.Another pr will split to volume level when this pr is merge.

Design doc
https://drive.google.com/file/d/1eaE8vSEhIli0H3j2eDiPJNYuKAC0MFgu/view?usp=sharing
https://issues.apache.org/jira/browse/HDFS-15382
https://issues.apache.org/jira/browse/HDFS-15382

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 16, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 17s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 13 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 13m 4s Maven dependency ordering for branch
+1 💚 mvninstall 24m 52s trunk passed
+1 💚 compile 24m 15s trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 compile 20m 45s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 3m 55s trunk passed
+1 💚 mvnsite 3m 15s trunk passed
+1 💚 javadoc 2m 20s trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 3m 21s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 6m 3s trunk passed
+1 💚 shadedclient 26m 37s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for patch
+1 💚 mvninstall 2m 24s the patch passed
+1 💚 compile 23m 40s the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
-1 ❌ javac 23m 40s /results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 6 new + 1862 unchanged - 5 fixed = 1868 total (was 1867)
+1 💚 compile 20m 45s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌ javac 20m 45s /results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu120.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu120.04-b07 generated 6 new + 1738 unchanged - 5 fixed = 1744 total (was 1743)
+1 💚 blanks 0m 1s The patch has no blanks issues.
-0 ⚠️ checkstyle 3m 48s /results-checkstyle-root.txt root: The patch generated 14 new + 554 unchanged - 5 fixed = 568 total (was 559)
+1 💚 mvnsite 3m 12s the patch passed
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
-1 ❌ javadoc 1m 9s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.
+1 💚 javadoc 3m 21s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌ spotbugs 3m 45s /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 💚 shadedclient 26m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 17m 35s hadoop-common in the patch passed.
+1 💚 unit 371m 23s hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
610m 51s
Reason Tests
SpotBugs module:hadoop-hdfs-project/hadoop-hdfs
Should org.apache.hadoop.hdfs.server.common.NoLockManager$NoLock be a static inner class? At NoLockManager.java:inner class? At NoLockManager.java:[lines 29-41]
org.apache.hadoop.hdfs.server.datanode.DataSetLockManager.lockLeakCheck() makes inefficient use of keySet iterator instead of entrySet iterator At DataSetLockManager.java:keySet iterator instead of entrySet iterator At DataSetLockManager.java:[line 258]
Should org.apache.hadoop.hdfs.server.datanode.DataSetLockManager$TrackLog be a static inner class? At DataSetLockManager.java:inner class? At DataSetLockManager.java:[lines 99-128]
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/artifact/out/Dockerfile
GITHUB PR #3889
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname Linux 92f3788f2864 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 21ab47a
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/testReport/
Max. process+thread count 1976 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@Hexiaoqiao
Copy link
Contributor

Hexiaoqiao commented Jan 17, 2022

Thanks @MingXiangLi for your works. It is a great progress now.
Take a quick glance, I think it is better to split it to different separate JIRA and PR, and more smooth to merge.
cc @jojochuang @sodonnel @goiri @ayushtkn Would you mind to take a look and give some other suggestions? Thanks.

@MingXiangLi MingXiangLi mentioned this pull request Jan 18, 2022
Merged
@MingXiangLi
Copy link
Contributor Author

MingXiangLi commented Jan 18, 2022

感谢@MingXiangLi的作品。现在是一个很大的进步。 快速浏览一下，我认为最好将其拆分为不同的独立JIRA和PR，并且合并更顺畅。 cc @jojochuang @sodonnel @goiri @ayushtkn你介意看看并提供一些其他建议吗？谢谢。

Good advice. Add sub task to this pr. #3900

@Hexiaoqiao
Copy link
Contributor

Hexiaoqiao commented Jan 23, 2022

Close this PR. Please trace https://issues.apache.org/jira/browse/HDFS-15382 or #3900 to follow the new progress if interested.

@Hexiaoqiao Hexiaoqiao closed this Jan 23, 2022
@xinglin
Copy link
Contributor

xinglin commented Sep 2, 2023

@MingXiangLi are these 30K DNs in the same cluster or multiple clusters? What is the largest single cluster size you have run/tested?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@MingXiangLi @hadoop-yetus @Hexiaoqiao @xinglin